Add harbor_env — serve Harbor task directories as an OpenEnv environment - #1018
Add harbor_env — serve Harbor task directories as an OpenEnv environment#1018thegovind wants to merge 9 commits into
harbor_env — serve Harbor task directories as an OpenEnv environment#1018Conversation
…Env env Adds envs/harbor_env, which runs any Harbor task directory (https://www.harborframework.com/docs/tasks) as a Gymnasium-style OpenEnv environment. The task directory is the interface: a directory that `harbor run` accepts is served unchanged — no conversion step, no second copy of the data, and no producer-specific code in OpenEnv. That covers Terminal-Bench-lineage tasks and everything Repo2RLEnv generates from a GitHub repository, so a task set built for evaluation is also a training environment. Design ------ The environment is deliberately thin and split by concern: task.py parses task.toml and discovers task directories sandbox.py Harbor's sandbox contract; local and docker backends reward.py Harbor's reward-file contract harbor_env_environment.py reset/step/state over the three above app.py FastAPI Rewards are always produced by the task's own tests/test.sh and only forwarded, per OpenEnv's rewards-in-environment invariant. reward.json is read first and reward.txt is the fallback, matching Harbor. A verifier that writes no reward file yields reward=None and an explicit error rather than a fabricated 0.0, which would be indistinguishable from a genuine failure and would poison training data. exec/read/write are the agent's action space; evaluate and solve are orchestration controls and are never exposed to the agent, keeping the dual API boundary intact. read and write are confined to the working directory so an agent cannot reach /tests or /solution, and the verifier log directory is cleared immediately before the verifier runs so a planted reward file cannot survive into the score. Two backends: `local` runs subprocesses under a per-episode directory tree with Harbor's paths exported as HARBOR_* variables (no Docker, so it works on HF Spaces), and `docker` runs inside the task's own image, which is required for tasks whose starting state lives in an image. Local mode refuses image-backed tasks with an actionable message instead of grading an empty directory, and is documented as a filesystem boundary rather than a security one. Task sets load from a local directory or straight from a Hugging Face dataset (hf://datasets/<org>/<name>[@rev]), including the tasks/<id>/ layout that `repo2rlenv push` writes. Verification ------------ The bundled examples/tasks/fix-sum-bug task is written to run under every runtime and grades identically on all three — 0.600 untouched, 1.000 after solution/solve.sh — under `harbor run` (harbor 0.20.0), harbor_env local mode, and harbor_env docker mode. A real Repo2RLEnv-emitted pr_runtime task (produced by repo2rlenv's own emitter: version = "1.0", [metadata.repo2env], environment/Dockerfile with WORKDIR /workspace, tests/test.sh writing /logs/verifier/reward.txt) scores 0.0 for a no-op agent and 1.0 for the oracle under both `harbor run` and harbor_env docker mode, with reward-details.json surfaced intact as observation.info["reward_details"]. tests/envs/test_harbor_env.py adds 50 hermetic tests (no Docker, no network) covering task parsing including Repo2RLEnv's `version` spelling, catalog discovery, the reward-file precedence chain, workdir escape rejection, the agent/orchestration boundary, and the client/server wire contract. Full suite: 1463 passed, 130 skipped. Also adds examples/quickstart.py, examples/validate_taskset.py (an oracle sweep to run before training on a generated task set), the docs stub, and the environment catalog entry.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Pull request overview
Adds a new self-contained OpenEnv environment, envs/harbor_env, that serves Harbor task directories (including Repo2RLEnv-emitted tasks) as reset/step episodes with verifier-produced rewards forwarded from Harbor’s reward-file contract.
Changes:
- Introduces
harbor_envserver/client/models, plus task discovery/parsing, sandbox backends (local + Docker), and reward parsing. - Ships a bundled end-to-end example Harbor task (
fix-sum-bug) and runnable validation/quickstart scripts. - Adds docs and catalog entries for the new environment.
Alignment Review Report
Automated Checks
- Lint: UNKNOWN — I can’t execute
bash .claude/hooks/lint.shin this review context. - Debug code: FOUND —
src/contains existingprint(...)statements (e.g.src/openenv/core/mcp_client.py) and existing TODOs; the hook is informational (exits 0) but will report these.
Open RFCs Context
- In Review: RFC 000, 001, 002, 003, 005
- Draft: RFC 010
No direct conflicts identified with the Harbor environment implementation; this PR mostly introduces a new environment rather than changing core contracts.
Tier 1: Fixes Required
- envs/harbor_env/server/sandbox.py:489-496 —
DockerSandbox.upload_dir()builds tar archives with duplicate entries due totarfile.add()default recursion (stored PR comment ID: 001). - envs/harbor_env/server/harbor_env_environment.py:243-251 —
_do_read()should reject empty paths with a clear error (stored PR comment ID: 002). - envs/harbor_env/server/harbor_env_environment.py:253-261 —
_do_write()should reject empty paths with a clear error (stored PR comment ID: 003).
Tier 2: Alignment Discussion
Principle Conflicts
ALIGNMENT FLAG: Orchestration-only controls (evaluate, solve) live in the same action schema as agent actions.
- Principle/RFC at stake: System invariant “Agent isolation” / “Dual API boundary” (INVARIANTS.md)
- The concern: The code relies on the trainer/policy wiring to keep
evaluate/solveout of the agent’s action space. If a policy is allowed to choose freely among action types, it can end episodes or apply the oracle solution. - Suggested reviewer:
@darktex
ALIGNMENT FLAG: local mode executes arbitrary shell commands as the server user (no container boundary).
- Principle/RFC at stake: “Container isolation” (PRINCIPLES.md / INVARIANTS.md)
- The concern: This is documented as “not a security boundary,” but it’s still a sharp edge worth explicitly validating as acceptable for a new environment (especially for deployment defaults).
- Suggested reviewer:
@darktex
RFC Conflicts
None identified.
Show a summary per file
| File | Description |
|---|---|
| tests/envs/test_harbor_env.py | Adds hermetic unit + end-to-end tests for task parsing, reward semantics, sandbox confinement, and client/server wire compatibility. |
| envs/harbor_env/server/task.py | Implements Harbor task parsing, schema compatibility (schema_version vs version), discovery via TaskCatalog, and task-source resolution (local or HF dataset). |
| envs/harbor_env/server/sandbox.py | Adds sandbox contract + implementations for local subprocess execution and Docker execution, including task staging and verifier/oracle execution. |
| envs/harbor_env/server/reward.py | Implements Harbor reward-file precedence (reward.json then reward.txt) and optional Repo2RLEnv sidecar surfacing. |
| envs/harbor_env/server/harbor_env_environment.py | Orchestrates reset/step lifecycle over tasks + sandbox, forwarding verifier rewards into observations/state. |
| envs/harbor_env/server/Dockerfile | Builds a runnable server image for harbor_env with uv/venv setup and runtime deps. |
| envs/harbor_env/server/app.py | FastAPI app wiring via openenv server utilities and env factory configuration. |
| envs/harbor_env/server/init.py | Exposes server-side public API surface (Environment, Sandbox backends, parsing helpers). |
| envs/harbor_env/README.md | Environment README/Space metadata, usage, format contract, and operational guidance. |
| envs/harbor_env/pyproject.toml | Defines the env package, deps (incl. docker SDK), and the server script entry point. |
| envs/harbor_env/openenv.yaml | Registers the environment for OpenEnv tooling (name/version/runtime/types). |
| envs/harbor_env/models.py | Defines shared Pydantic wire models (action/observation/state) for client/server separation. |
| envs/harbor_env/client.py | Implements HarborEnv client wrappers and result/state parsing. |
| envs/harbor_env/init.py | Package exports for client + wire types. |
| envs/harbor_env/examples/validate_taskset.py | Provides an oracle-sweep validator to verify task solvability before training. |
| envs/harbor_env/examples/quickstart.py | Demonstrates a complete solve-and-evaluate episode against a running server. |
| envs/harbor_env/examples/tasks/fix-sum-bug/task.toml | Bundled example Harbor task config used for end-to-end validation. |
| envs/harbor_env/examples/tasks/fix-sum-bug/instruction.md | Example task instruction prompt served on reset. |
| envs/harbor_env/examples/tasks/fix-sum-bug/environment/stats.py | Seeded “buggy” starting state for the example task. |
| envs/harbor_env/examples/tasks/fix-sum-bug/tests/test.sh | Verifier entrypoint that runs the grader and writes reward artifacts. |
| envs/harbor_env/examples/tasks/fix-sum-bug/tests/grade.py | Standard-library grader that computes partial credit and writes reward.json + reward.txt. |
| envs/harbor_env/examples/tasks/fix-sum-bug/solution/solve.sh | Oracle script that installs the reference solution into the workdir. |
| envs/harbor_env/examples/tasks/fix-sum-bug/solution/stats.py | Reference solution implementation used by the oracle. |
| docs/source/environments/harbor.md | Adds official docs page for the new Harbor environment. |
| docs/source/environments.md | Adds Harbor to the environments catalog page. |
| docs/source/_toctree.yml | Adds Harbor docs page to the documentation toctree. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (1)
envs/harbor_env/server/harbor_env_environment.py:261
_do_write()doesn’t validate thataction.pathis non-empty. Since the model default is "", this can resolve to the workdir and then attempt to write to a directory path, producing an opaque backend exception. Reject empty/whitespace-only paths up front to keep the action contract clear and errors stable across backends.
def _do_write(
self, action: HarborAction, sandbox: Sandbox, task: HarborTask
) -> HarborObservation:
del task
target = resolve_within(sandbox.paths.workdir, action.path)
sandbox.write_text(target, action.content)
return self._observe(
"write", output=f"wrote {len(action.content)} bytes to {action.path}"
)
- Files reviewed: 26/27 changed files
- Comments generated: 2
- Review effort level: Low
…ion, empty paths Cursor Bugbot and Copilot both reviewed huggingface#1018. Four concrete findings, all real: 1. reset() left the previous episode in state (Bugbot, medium) close() tears the sandbox down first, but _state was only replaced after catalog.get() and sandbox.start(), either of which can raise. A failed reset therefore reported a task_id, workdir and reward for an episode that was no longer running. Now drops to a clean HarborState before the fallible work. 2. DockerSandbox.upload_dir() archived nested files twice (Copilot) rglob("*") yields directories as well as files and tarfile.add() recurses by default, so every nested file was added once via its parent and again on its own iteration. Passes recursive=False; directory entries are still emitted, so empty directories survive. 3/4. read/write accepted an empty path (Copilot; write was filed low-confidence) HarborAction.path defaults to "", which resolve_within() happily resolved to the working directory itself. For read that produced a confusing "no such file: ". For write it was destructive: the target split into ("/", "workspace"), so the upload would have dropped a regular file over the whole checkout. Fixed in resolve_within() rather than at the two call sites, since that is the single chokepoint every agent-supplied path passes through — "", " ", "." and "a/.." are all rejected now. Also, from Copilot's Tier 2 flag that evaluate/solve share the action schema with agent actions: AGENT_ACTIONS and CONTROL_ACTIONS were decorative — exported but never referenced. HarborActionType is now composed from AgentActionType and ControlActionType instead of restating the five strings, so a new action has to be classified as one or the other to exist at all, and a test asserts the two sets remain a partition of the handler table. That does not resolve the boundary question itself, which is left for human review. Tests: 8 added (58 total). Each regression test was confirmed to fail against the unfixed code before being kept. Full suite 1471 passed, 130 skipped.
|
Thanks both — all four findings were real, and they're fixed in f568159. Each regression test was confirmed to fail against the unfixed code before I kept it, so none of these can quietly come back. Tier 11. Failed reset left the previous episode in 2. 3/4. Empty Tier 2 — alignment flags
Partially addressed: That kills the drift risk but does not resolve the boundary question, so I've left it open for @Darktex. For context on where I think it actually stands:
Full suite after the fixes and the |
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (3)
envs/harbor_env/server/harbor_env_environment.py:211
- After
evaluateends the episode (done=Trueandstate.evaluated=True),step()still allows further actions because_require_episode()continues to succeed. This contradicts the one-shot episode contract and can lead to trainers accidentally continuing to mutate a finished episode.
self._state.step_count += 1
self._state.last_action_type = action.action_type
try:
sandbox, task = self._require_episode()
return self._handlers[action.action_type](action, sandbox, task)
envs/harbor_env/server/task.py:197
- The
needs_imagedocstring says anydocker_imageimplies the starting state lives in an image, but the implementation treats tasks withenvironment/seed files as not needing Docker even ifdocker_imageis set. Please update the docstring so it matches the actual semantics (seed files => local mode allowed).
A task that ships seed files carries its own starting state and can be
reproduced anywhere. Anything else (a `Dockerfile`, a compose file, or a
pre-built `docker_image`) puts the starting state in an image, so only
the Docker execution mode can run it faithfully.
envs/harbor_env/models.py:79
HarborObservation.successis documented as being independent of command exit status ("a failing test command is a successful exec"), but the implementation/tests setsuccess=Falsewhenexecexits non-zero. This docstring should be corrected to match the actual wire semantics so downstream consumers interpretsuccesscorrectly.
success (`bool`):
Whether the action itself completed. Unrelated to the reward: a
failing test command is a successful `exec`.
- Files reviewed: 26/27 changed files
- Comments generated: 0 new
- Review effort level: Low
…enEnv PR Automated reviewers went over huggingface/OpenEnv#1018, which carries a structurally parallel runtime. All four findings applied here too — this repo just had no bot looking at it. 1. reset() left the previous episode in state close() tears the container down first, but _state was only replaced after tasks.get() and sandbox.start(), either of which can raise. A failed reset reported a task_id, workdir and reward for an episode that was gone. Now drops to a clean Repo2RLEnvState before the fallible work. 2. DockerSandbox.upload_dir() archived nested files twice rglob("*") yields directories as well as files and tarfile.add() recurses by default, so anything nested was added once via its parent and again on its own iteration. Passes recursive=False; directory entries are still emitted, so nested staging (e.g. the tests/verifier.py + tests/*.json layout our pipelines emit as aux_files) keeps working. 3. Degenerate paths resolved to the checkout itself Repo2RLEnvAction.path defaults to "", and resolve_within() resolved "", ".", " " and "a/.." to the working directory. For read that meant catting a directory; for write it was destructive — the target split into ("/", "workspace"), so an empty path would have dropped a regular file over the entire checkout. Rejected in resolve_within(), the single chokepoint every agent-supplied path passes through. 4. AGENT_ACTIONS / CONTROL_ACTIONS were decorative Exported but never referenced, and the test asserting them merely restated the same literals, so it would have passed no matter what the server handled. ActionType is now composed from AgentActionType | ControlActionType rather than restating the five strings, and the test asserts the two sets are a partition of the environment's actual handler table. Verification: each regression test was confirmed to fail against the unfixed code before being kept. Also re-ran the Docker end-to-end with a task whose tests/ has a nested subdirectory — the verifier reads tests/helpers/score.py, proving recursive=False did not break nested staging — and confirmed an empty write is now rejected with the checkout left intact. Tests: 39 (6 added). Full suite 739 passed.
|
Three minor issues for completeness:
|
…isode terminal Addresses the three issues raised in review, plus two related holes found while verifying them. ## [p1] local mode was the default and leaked too much - `HARBOR_MODE` now defaults to `docker`. It is the faithful backend and the only one that can enforce a task's declared policies; `local` is an explicit opt-in for Docker-less hosts such as Spaces. - `LocalSandbox.exec` no longer hands subprocesses the server's whole environment. Task commands are attacker-controlled input, so they now start from a short allowlist (PATH, HOME, LANG, …) and the process's API keys and Hub tokens stay out. A task that needs a secret declares it in `[environment].env`, which is resolved explicitly. While confirming the escape, two further leaks turned up: - `reset()` put the task's server-side directory in the observation, which reaches the agent. In local mode `exec cat <path>/solution/…` then read the oracle outright. `HarborTask.summary()` no longer carries `path`; `HarborState.task_path` still does, since state is infrastructure-side. - `HARBOR_TESTS_DIR` / `HARBOR_SOLUTION_DIR` / `HARBOR_LOGS_DIR` were exported into agent commands. `SandboxPaths.as_env(agent_visible=True)` now withholds them; the verifier and oracle still receive the full set. - `stage_tests` / `stage_solution` now wipe the destination first. The agent shares the filesystem and could pre-create `/tests`; a planted file survived staging and was read by the verifier. Regression test confirms it scored 1.0 before this change. ## [p1] network, resource and user policies were ignored `task.py` now parses `[environment].network_mode` / `allowed_hosts` / `cpus` / `memory_mb` / `storage_mb` / `gpus` / `workdir` and `[agent].user` / `[verifier].user`, including the deprecated `allow_internet` spelling. - Docker: `no-network` becomes `network_mode="none"`, cpus/memory become real container limits, and each phase runs as its declared user. Verified against a live container — `getent hosts pypi.org` fails and `memory.max` reads 536870912. - Unenforceable policies are refused rather than approximated: `allowlist` (needs a filtering proxy) and `gpus`. An unknown `network_mode` is a parse error, so it can never silently degrade to `public`. - Phases that disagree get the restrictive setting — a container has one network, and resolving that conflict permissively would hand a sandboxed verifier the internet. - Local: refuses any task declaring a policy it cannot enforce, naming HARBOR_MODE=docker, instead of running it unconstrained and reporting a score the task never sanctioned. ## [p2] evaluate/solve split unenforced; done walked back - After `evaluate` the episode is terminal: every later action is refused and keeps reporting `done=True`. Previously a post-evaluate `exec` returned `done=False`, contradicting a terminal state already reported to the trainer. - `allow_control_actions` (env `HARBOR_ALLOW_CONTROL_ACTIONS=0`) makes the server refuse `evaluate`/`solve` outright, so the agent/orchestration split is enforced rather than left to the caller's wiring. Left on by default, since a run that cannot score itself is only useful for agent-facing deployments. Tests: 14 added (72 total). Each regression test was confirmed to fail against the unfixed code. Existing tests now pass `mode="local"` explicitly rather than relying on the old default. Full suite 1578 passed, 131 skipped.
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (4)
envs/harbor_env/server/task.py:634
_positive_float()currently acceptsboolvalues becauseboolis a subclass ofintin Python (e.g.Truebecomes1.0). This can silently treat malformedtimeout_sec/ policy values as valid.
def _positive_float(value: Any, fallback: float) -> float:
try:
parsed = float(value) # type: ignore[arg-type]
except (TypeError, ValueError):
return fallback
return parsed if parsed > 0 else fallback
envs/harbor_env/server/harbor_env_environment.py:103
- Docstring says
modedefaults to$HARBOR_MODEthen "local", but the implementation uses DEFAULT_MODE="docker". This mismatch can mislead users about the default execution backend.
mode (`str`, *optional*):
Sandbox backend, `"local"` or `"docker"`. Defaults to `$HARBOR_MODE`,
then `"local"`.
envs/harbor_env/server/task.py:533
resolve_task_source()treats any existing path as a valid task source, even if it is a file. This can lead to confusing downstream errors (empty catalog) instead of failing fast with an actionable message.
local = Path(text).expanduser()
if local.exists():
return local.resolve()
envs/harbor_env/server/task.py:626
_positive_int()currently acceptsboolvalues becauseboolis a subclass ofintin Python (e.g.Truebecomes1). This can silently mis-parse malformedtask.tomlpolicy values instead of rejecting them.
This issue also appears on line 629 of the same file.
def _positive_int(value: Any) -> int | None:
try:
parsed = int(value) # type: ignore[arg-type]
except (TypeError, ValueError):
return None
return parsed if parsed > 0 else None
- Files reviewed: 26/27 changed files
- Comments generated: 0 new
- Review effort level: Low
A maintainer reviewed huggingface/OpenEnv#1018, which carries a structurally parallel runtime, and raised three issues. Two apply here in full; the third (local-mode escapes) does not, since this runtime is Docker-only. ## Task policies were ignored `dataset.py` now parses `[environment].network_mode` / `cpus` / `memory_mb` / `gpus` / `workdir` and `[agent].user` / `[verifier].user`, including the deprecated `allow_internet` spelling. `_container_limits()` translates them into `containers.run` kwargs: no-network becomes an isolated network, cpus/memory become real container limits, and each phase runs as its declared user. What cannot be enforced faithfully is refused rather than approximated — `allowlist` needs a filtering proxy and `gpus` needs accelerators, so both point the caller at `harbor run`. An unrecognized network_mode is a parse error, never a silent downgrade to `public`: quietly granting a sandboxed task the internet is the exact failure this prevents. Our pipelines do not emit these fields today, but the format allows them and a task that declares one was being run unconstrained and scored anyway. ## The episode never terminated, and the action split was decorative - After `evaluate`, `step()` refused nothing: a later `exec` returned `done=False`, contradicting a terminal state already reported to the caller. The episode is now terminal until `reset()`. - `allow_control_actions` (env `REPO2RLENV_ALLOW_CONTROL_ACTIONS=0`) makes the server refuse `evaluate`/`solve`, so the agent/orchestration split is enforced rather than left to the caller's wiring. ## Two leaks found while verifying the above - `HARBOR_TESTS_DIR` / `HARBOR_SOLUTION_DIR` / `HARBOR_LOGS_DIR` were exported into agent commands, handing a policy the location of the grading logic and the answer. `SandboxPaths.as_env(agent_visible=True)` withholds them; the verifier and oracle still receive the full set. - `stage_tests` / `stage_solution` now wipe the destination first. The agent shares the container and can pre-create `/tests`; anything planted there previously survived staging and was read by the verifier. Tests: 6 added (45 total). Full suite 740 passed. `tests/test_e2e_public.py:: test_e2e_public_trl` fails on a clean tree too — it queries live GitHub and TRL's two most recent PRs are currently docs-only.
Self-review of the policy work found that honouring [agent].user broke every task that declares one. The Harbor directories are created by the image's default account, so an agent running as `nobody` inherited a working directory it could not write to: `touch probe.txt` returned Permission denied, and every `write` action would have failed. The same applied to [verifier].user and /logs/verifier, which would have left those tasks unscorable. Sandbox.chown() now hands the working directory and agent log dir to the agent user at start, and the verifier log and tests dirs to the verifier user before the verifier runs. Verified against a live container: the agent is `nobody`, the workdir is writable, and the task still grades 0.6. Also notes that storage_mb needs a quota-capable storage driver; where one is absent Docker rejects the container, which is the fail-closed outcome.
|
Thanks Ben — all three were real, and hunting them turned up three more. Fixed across [p1] local mode default and its blast radiusAgreed, and the escape was worse than described. obs = env.reset(task_id="fix-sum-bug")
env.step(HarborAction(action_type="exec",
command=f"cat {obs.info['task']['path']}/solution/stats.py"))
# → the oracle, in fullThat is a complete reward-hacking hole, not just an isolation weakness. Changes:
[p1] network / resource / user policies ignoredCorrect — parsed and enforced now, against Harbor's own schema (
Verified against a live container, not just unit-tested: Two judgement calls worth your eyes:
[p2] split unenforced, and
|
|
|
||
|
|
||
| @lru_cache(maxsize=None) | ||
| def resolve_task_source(source: str | os.PathLike[str] | None) -> Path: |
There was a problem hiding this comment.
PathLike task source crashes
Medium Severity
resolve_task_source is wrapped in lru_cache but accepts os.PathLike values such as pathlib.Path. Path objects are not hashable, so a normal call like resolve_task_source(Path("./tasks")) raises TypeError instead of resolving the directory, despite the documented API.
Reviewed by Cursor Bugbot for commit 270a457. Configure here.
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (2)
envs/harbor_env/server/harbor_env_environment.py:103
- The docstring says the
modedefault falls back to "local", but the implementation usesDEFAULT_MODE = "docker"whenHARBOR_MODEis unset. This mismatch can mislead users reading the API docs and contradictsapp.py/README which document docker as the default.
mode (`str`, *optional*):
Sandbox backend, `"local"` or `"docker"`. Defaults to `$HARBOR_MODE`,
then `"local"`.
envs/harbor_env/server/task.py:533
resolve_task_source()treats any existing path as a valid task root, including regular files. If a file path is passed (by typo or env var), this will silently resolve and later yield an empty catalog instead of failing fast with an actionable error.
local = Path(text).expanduser()
if local.exists():
return local.resolve()
- Files reviewed: 26/27 changed files
- Comments generated: 0 new
- Review effort level: Low
| self.dockerfile is not None | ||
| or self.compose_file is not None | ||
| or bool(self.docker_image) | ||
| ) |
There was a problem hiding this comment.
Local mode ignores docker_image
Medium Severity
When a task has environment/ seed files and a [environment].docker_image, needs_image returns false, so local mode runs on the host and never uses the declared image. Harbor still runs those seeds inside that image; local grading can fail or score incorrectly when the verifier or agent depends on packages only in the image.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 96bc040. Configure here.
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (2)
envs/harbor_env/models.py:80
HarborObservation.successis documented as being independent of command exit status, but the server setssuccessbased onExecResult.ok(exit code 0 and not timed out). The docstring should match the actual wire semantics so clients/tests interpret it consistently.
success (`bool`):
Whether the action itself completed. Unrelated to the reward: a
failing test command is a successful `exec`.
error (`str`):
envs/harbor_env/server/harbor_env_environment.py:103
- Docstring says
modedefaults to"local", but the implementation usesDEFAULT_MODE = "docker"whenHARBOR_MODEis unset. This can mislead users about the default execution backend.
mode (`str`, *optional*):
Sandbox backend, `"local"` or `"docker"`. Defaults to `$HARBOR_MODE`,
then `"local"`.
- Files reviewed: 26/27 changed files
- Comments generated: 0 new
- Review effort level: Low
…ardening A rubber-duck pass over 592de2b/270a4579 found that several of those fixes were incomplete, and that one of them had silently defanged an existing test. ## The oracle was still reachable in local mode Removing `path` from `HarborTask.summary()` was not enough: `HarborState` carried `task_path`, and `state` rides the same socket as `step`. A caller that could act could still read the host path and `cat <path>/solution/…`. The field is gone. ## read/write bypassed [agent].user entirely `read_text()` ran `cat` as the image's default account and `write_text()` extracted a root-owned tar through the archive API, so a task declaring an unprivileged agent still got root's reach through those two actions. Demonstrated before the fix: a symlink in the working directory plus `read` returned `/etc/shadow`, and plus `write` created a root-owned file under `/root`. Both now run as the declared user, with writes going through the shell so the kernel applies its permissions. Re-verified against a live container — both attempts are refused and agent-written files are owned by `nobody`. ## Failed evaluation left the tests exposed and the episode running `evaluated` was set only after staging, chowning and executing, so a failure in between left `/tests` readable with `done=False`. The episode now terminates before anything that can fail. ## storage_mb was not fail-closed Docker accepts `storage_opt` on drivers that never enforce it — the review demonstrated a 64 MiB write inside a 32 MiB "limit". A task that believes it is capped and is not is worse than one told up front we cannot cap it, so this is now refused like `allowlist` and `gpus`. ## Malformed limits read as "no limit" `_positive_int` turned `0`, `-4` and `"lots"` into `None`, which meant both an unconstrained container *and* no refusal from the local backend. Invalid values are now `TaskFormatError`. ## Phase overrides did not replace the baseline `NetworkPolicy.modes` unioned the baseline unconditionally, so a task whose agent and verifier both declared `public` over a `no-network` baseline had its network disabled anyway. Only the effective per-phase modes count now. ## An existing security test had gone vacuous `test_planted_reward_files_are_discarded` planted through `$HARBOR_LOGS_DIR`, which 592de2b hid from agent commands — so the plant silently failed and the test passed for the wrong reason. It now uses the real path and asserts the plant landed before checking it is discarded. ## Smaller - Agent-supplied `timeout_s` is capped at the configured ceiling instead of overriding it; it arrives on the wire and was unbounded. - `validate_taskset.py` passes `allow_control_actions=True` — it *is* the orchestration, and was inheriting the deployment's flag. - The README no longer claims local mode keeps secrets from tasks. It keeps them out of the environment, but `/proc/<ppid>/environ` still exposes them; that is hygiene, not containment, and is now described as such. Tests: 78 (5 added). Full suite 1584 passed, 131 skipped.
|
Follow-up: I ran an independent review pass over my own fixes above, and several of them were incomplete. Fixed in Worth reading if you're reviewing the security claims, because one of my fixes silently defanged an existing test. The oracle was still reachable in local modeRemoving
|
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (3)
envs/harbor_env/server/sandbox.py:646
- DockerSandbox.write_text(): when
useris set, the implementation writes via shell redirection but does not create parent directories first (unlike the root/put_archive path and LocalSandbox). Writing to a nested path likesubdir/file.txtwill fail even thoughSandbox.write_text()promises to auto-create parents.
def write_text(self, path: str, content: str, user: str | None = None) -> None:
if user:
# put_archive extracts as root, which would let an unprivileged
# agent create root-owned files — and, through a symlink out of the
# working directory, create them anywhere. Write through the shell
envs/harbor_env/server/harbor_env_environment.py:103
- Docstring mismatch: the constructor defaults
modeto$HARBOR_MODEand thenDEFAULT_MODE(which is "docker"), but the docstring says it falls back to "local". This is user-facing API documentation for the environment.
mode (`str`, *optional*):
Sandbox backend, `"local"` or `"docker"`. Defaults to `$HARBOR_MODE`,
then `"local"`.
envs/harbor_env/server/task.py:397
- HarborTask.summary() docstring references
HarborState.task_path, but HarborState does not define such a field (and tests assert task directory paths are not leaked via state). This comment is misleading for maintainers.
Deliberately omits [`path`]: the observation reaches the agent, and
naming the task directory on the server would point straight at
`solution/` and `tests/`. [`HarborState.task_path`] still carries it for
training orchestration, which is on the infrastructure side.
- Files reviewed: 26/27 changed files
- Comments generated: 1
- Review effort level: Low
| if not target.startswith(base_norm.rstrip("/") + "/"): | ||
| raise ValueError(f"path escapes the working directory: {relative!r}") | ||
| return target |
| except (TypeError, ValueError) as exc: | ||
| raise TaskFormatError( | ||
| f"[environment].{field} must be a positive integer, got {value!r}" | ||
| ) from exc |
There was a problem hiding this comment.
TOML floats break resource limits
Low Severity
_positive_int parses resource fields with int(str(value)), so legitimate TOML numeric values like cpus = 2.0 raise TaskFormatError instead of loading the task, even when the value is a whole number.
Reviewed by Cursor Bugbot for commit 99f473b. Configure here.
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (3)
envs/harbor_env/server/sandbox.py:650
- In
DockerSandbox.write_text, theuserbranch writes via shell redirection but never ensures the parent directory exists. This violates the method contract (“creating parent directories as needed”) and will fail for writes to nested paths (e.g.path="/app/src/foo.py"). Create the parent directory before executing the write command (same as the put_archive branch).
def write_text(self, path: str, content: str, user: str | None = None) -> None:
if user:
# put_archive extracts as root, which would let an unprivileged
# agent create root-owned files — and, through a symlink out of the
# working directory, create them anywhere. Write through the shell
envs/harbor_env/server/harbor_env_environment.py:103
- The
modeparameter docstring says the default falls back to"local", but the implementation usesDEFAULT_MODE = "docker"whenHARBOR_MODEis unset. This is user-facing documentation and should match the actual default behavior.
mode (`str`, *optional*):
Sandbox backend, `"local"` or `"docker"`. Defaults to `$HARBOR_MODE`,
then `"local"`.
envs/harbor_env/server/Dockerfile:52
- This comment says
timeoutbounds commands in the local backend, but the local backend uses Python’ssubprocess.run(..., timeout=...). Thetimeoutbinary is actually used by the Docker backend (DockerSandbox._with_timeout) inside the task container. Updating the comment will avoid misleading operators about where command bounding happens.
# Harbor verifiers commonly shell out to git, bash and coreutils; `timeout`
# bounds commands in the local backend's subprocesses, and curl backs HEALTHCHECK.
RUN apt-get update && \
- Files reviewed: 26/27 changed files
- Comments generated: 0 new
- Review effort level: Low
Cursor and Copilot both flagged the same hole independently, and they were
right: `resolve_within()` is a purely lexical check, so it cannot see a symlink
the agent planted with `exec`. Demonstrated before the fix:
exec ln -s <tests dir> link
read path="link/grade.py" -> returned the grader's source
That defeats the documented guarantee that `/tests` and `/solution` are
unreachable through `read`/`write` — the agent could read the grading logic, and
`write` could plant files the verifier would then execute.
Agent paths now go through `Sandbox.resolve_agent_path()`, which keeps the
lexical check and then canonicalizes on the sandbox's own filesystem:
`Path.resolve()` locally and `readlink -m` in the container. `-m` rather than
`-f` because a `write` may legitimately create directories that do not exist
yet, while symlinks that *do* exist are still resolved — which is the only part
confinement depends on.
The base is canonicalized too. Without that the fix rejected everything on
macOS, where the episode root lives under `/var`, itself a link to
`/private/var`. Both cases are covered by tests.
Also, from the same round of review:
- Local mode now flags a task that declares `[environment].docker_image` while
running on the host toolchain. Seed files make the *state* reproducible, but
the image supplies the toolchain, and a version skew would otherwise surface
as a mis-graded episode. Warned in the log and surfaced in the reset
observation as `info["toolchain_warning"]`.
Two other items from that round needed no change:
- "Failed evaluate leaves the episode open" was reported against 270a457 and
was already fixed in dd2232b, which terminates before the verifier runs.
Verified: a task with no `tests/test.sh` returns `done=True`, as does every
action after it.
- "PathLike task source crashes" does not reproduce. `pathlib.Path` is hashable,
so `lru_cache` accepts it; `resolve_task_source(Path("..."))` resolves
normally.
Tests: 81 (3 added). The symlink test was confirmed to fail against the
pre-fix lexical check. Full suite 1587 passed, 131 skipped.
|
Thanks both — the symlink finding was the real one, and it was a good catch. Fixed in Symlink escape from
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 4 total unresolved issues (including 3 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5079a77. Configure here.
| raise ValueError( | ||
| f"path escapes the working directory through a link: {relative!r}" | ||
| ) | ||
| return real |
There was a problem hiding this comment.
Windows workdir confinement broken
Medium Severity
resolve_agent_path decides confinement with real.startswith(base.rstrip("/") + "/"), which assumes POSIX trailing slashes. On Windows, real_path returns backslash paths, so legitimate files under the workdir often fail the prefix test and read/write incorrectly raise path-escape errors—or checks behave inconsistently. resolve_within uses the same slash logic on Windows workdir strings.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 5079a77. Configure here.
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (4)
envs/harbor_env/server/sandbox.py:717
- When writing as a declared user, DockerSandbox.write_text() redirects into the target path without creating parent directories. This breaks writes to nested paths (which the Sandbox.write_text contract allows) for tasks that set [agent].user / [verifier].user.
payload = base64.b64encode(content.encode("utf-8")).decode("ascii")
result = self.exec(
f"printf %s {_quote(payload)} | base64 -d > {_quote(path)}",
timeout_s=120.0,
user=user,
envs/harbor_env/server/harbor_env_environment.py:101
- Docstring says the default sandbox mode falls back to "local", but the implementation uses DEFAULT_MODE = "docker" when HARBOR_MODE is unset. This can mislead users reading the API docs.
mode (`str`, *optional*):
Sandbox backend, `"local"` or `"docker"`. Defaults to `$HARBOR_MODE`,
then `"local"`.
envs/harbor_env/models.py:80
- HarborObservation.success is documented as "Unrelated to the reward: a failing test command is a successful exec", but the server sets success based on ExecResult.ok (exit_code==0 and not timed out). The docstring should reflect the actual semantics so clients don't misinterpret failures.
success (`bool`):
Whether the action itself completed. Unrelated to the reward: a
failing test command is a successful `exec`.
error (`str`):
envs/harbor_env/server/sandbox.py:698
- DockerSandbox.read_text() currently returns None for any non-zero cat exit code (including permission errors or other failures). Callers treat None as "file missing" and will surface misleading FileNotFoundError messages; reward reading also loses useful diagnostics. Distinguish "missing" from other errors and raise on non-missing failures.
def read_text(self, path: str, user: str | None = None) -> str | None:
result = self.exec(f"cat {_quote(path)}", timeout_s=60.0, user=user)
return result.output if result.ok else None
- Files reviewed: 26/27 changed files
- Comments generated: 0 new
- Review effort level: Low


Summary
A new environment,
envs/harbor_env, that runs any Harbor task directory as a Gymnasium-style OpenEnv environment.The task directory is the interface. A directory that
harbor runaccepts is served here unchanged — no conversion step, no re-authoring, no second copy of the data, and no producer-specific code in OpenEnv. That covers Terminal-Bench-lineage tasks and everything Repo2RLEnv generates from a GitHub repository, so a task set built for evaluation is also a training environment.Where this sits
Harbor and OpenEnv are not competitors — they sit at different layers, and they agree on the one contract that matters: the reward is produced inside the environment and only forwarded.
flowchart LR P["Task producers<br/>Repo2RLEnv · Terminal-Bench<br/>· hand-written"] T["Harbor task directory<br/><i>the format</i>"] H["harbor run<br/><i>batch evaluation</i>"] E["envs/harbor_env<br/><b>NEW</b> — episode loop"] TR["Trainer / RL loop<br/>reset · step · state"] P -->|emit| T T --> H T -->|"served unchanged"| E E --> TR style E fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px style T fill:#e3f2fd,stroke:#1565c0tests/test.sh→/logs/verifierObservation.reward, produced inside the envharbor_envis the bridge, and it generalizes the existingenvs/tbench2_env/pattern. The one real behavioural difference: tbench2 derives pass/fail from the verifier's exit code, while Harbor rewards are graded floats, soharbor_envreads the value the verifier wrote.An episode
sequenceDiagram participant TR as Trainer <br/>(infrastructure) participant E as HarborEnvironment participant S as Sandbox <br/>(local or docker) participant V as tests/test.sh TR->>E: reset(task_id="fix-sum-bug") E->>S: boot backend, seed working directory E-->>TR: Observation(instruction=instruction.md) loop agent works TR->>E: step(exec / read / write) E->>S: run in the working directory S-->>E: output end TR->>E: step(evaluate) E->>S: clear /logs/verifier, stage tests/ -> /tests E->>V: bash /tests/test.sh V-->>S: reward.json / reward.txt E-->>TR: Observation(reward=0.83, done=True) Note over E,V: reward.json first, then reward.txt —<br/>exactly Harbor's precedence Note over TR,E: evaluate + solve are infrastructure-only,<br/>never in the agent's action spaceInvariants held
reset/step/statestay on the infrastructure side.evaluateandsolveare orchestration controls, not agent actions — an agent that could grade on demand could end its own episode, and one that couldsolvecould hand itself the answer.harbor_envforwards whattests/test.shwrote and never computes a reward. If the verifier wrote no reward file, the result isreward=Noneplus an explicit error — not a0.0. A fabricated zero is indistinguishable from a genuine failure and would poison a training run.client.pyimports nothing fromserver/.readandwritego throughresolve_within(), confined to the working directory, so/testsand/solutionare unreachable./logs/verifieris wiped immediately before the verifier runs, so a reward file planted duringexeccannot survive into the score.Two execution modes
flowchart TD A["reset(task_id)"] --> B{"Where is the task's<br/>starting state?"} B -->|"ships environment/ seed files<br/><i>self-contained</i>"| C["local mode OK<br/><i>subprocesses, no Docker —<br/>works on HF Spaces</i>"] B -->|"Dockerfile / compose /<br/>docker_image"| D{"HARBOR_MODE"} D -->|docker| E["run inside the task's<br/>own image"] D -->|local| F["refuse with an actionable message<br/><i>rather than grading an empty directory</i>"] style C fill:#e8f5e9,stroke:#2e7d32 style E fill:#e8f5e9,stroke:#2e7d32 style F fill:#ffebee,stroke:#c62828localmode is documented as a filesystem boundary, not a security boundary —execruns as the server's own user with the server's own environment. Serve task sets you trust there; usedockerfor anything else.Verification
The interop claim is the whole point of this PR, so it was measured rather than asserted.
The bundled
examples/tasks/fix-sum-bugtask is written to run under every runtime (it prefers$HARBOR_*variables and falls back to Harbor's absolute paths). It grades identically in all three:solution/solve.shharbor run -a nop/-a oracle(harbor 0.20.0)harbor_envlocalmodeharbor_envdockermodeReproduce row one with
harbor run -p envs/harbor_env/examples/tasks/fix-sum-bug -a nop, and the others withexamples/validate_taskset.py.Separately, a real Repo2RLEnv-emitted
pr_runtimetask — produced by repo2rlenv's own emitter, soversion = "1.0"rather than Harbor'sschema_version, plus[metadata.repo2env],WORKDIR /workspace, and atests/test.shwriting/logs/verifier/reward.txt— scored 0.0 for a no-op agent and 1.0 for the oracle under bothharbor runandharbor_envdocker mode, withreward-details.jsonsurfacing intact asobservation.info["reward_details"].Note
High Risk
New environment runs attacker-controlled shell in local or Docker sandboxes and forwards training rewards; mis-grading or weak isolation could affect RL data and host security.
Overview
Introduces
envs/harbor_env, a new OpenEnv environment that serves Harbor task directories unchanged (including Repo2RLEnv output) over the standard reset/step API, with a FastAPI/WebSocket server,HarborEnvclient, and wire types forexec/read/writeplus infrastructure-onlyevaluateandsolve.Execution is split between
docker(task images, policy enforcement) andlocal(subprocesses under a per-episode tree for Spaces); local mode refuses tasks that need images or policies it cannot enforce. Rewards are read from verifier artifacts (reward.jsonthenreward.txt, withreward-details.jsonininfo); missing files yieldreward=None, not0.0. Sandboxing limits agent I/O to the workdir, stages/testsonly at verify time, and clears verifier logs before grading.Also adds a bundled
fix-sum-bugexample task,quickstart/validate_tasksetscripts, Docker image defaults, and docs entries (environments/harbor, catalog card, toctree).Reviewed by Cursor Bugbot for commit 5079a77. Bugbot is set up for automated code reviews on this repo. Configure here.